-
Notifications
You must be signed in to change notification settings - Fork 9
Multiple file input #29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
… log type checking
…m project configuration
…ace for improved organization
…ed I/O statistics and module management
…ss line mapping in DXT_POSIX data handling
…hecks in Darshan data handling
…in Darshan data handling
…atistics in Darshan data handling
…and stragglers count in Darshan data handling
…nd HDF5 extension check
Required for Python 3.8 compatability
…ons in Darshan handling
…y with older PyDarshan versions
Avoids using `iterrows()` which implicitly casts Darshan record IDs to float64, potentially breaking file_map lookups for large IDs. Replaced with `itertuples()` to maintain original dtype and re-enabled `break` for early exit.
…e logs Adds Sample_1 and Sample_2 run configs to easily test `reporter.py` on benchmark Darshan logs using the drishti-io module environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR updates the project to support multiple file inputs by changing the parser argument from a single log path to a list of paths. It also refactors several modules—reporter, parser, and the Darshan handler—to utilize the new argument, adds development tool dependencies, and introduces various assertions for metric validation.
- Updated parser: renamed “log_path” to “log_paths” with nargs="+"
- Refactored reporter and handle_darshan to work with multiple log inputs and utilize a DarshanFile object
- Added dev tools extras_require in setup.py and updated several formatting and assertion checks in the handler
Reviewed Changes
Copilot reviewed 5 out of 17 changed files in this pull request and generated 1 comment.
Show a summary per file
File | Description |
---|---|
setup.py | Added dev dependencies under extras_require |
drishti/reporter.py | Updated log type detection to work with lists and adjusted import patterns |
drishti/includes/parser.py | Changed positional argument from "log_path" to "log_paths" with improved flag formatting |
drishti/handlers/handle_darshan.py | Refactored handler to use DarshanFile, added assertions, and updated module check function calls |
.python-version | Set Python version to 3.8 |
Files not reviewed (9)
- .idea/.gitignore: Language not supported
- .idea/MypyPlugin.xml: Language not supported
- .idea/drishti-io.iml: Language not supported
- .idea/inspectionProfiles/profiles_settings.xml: Language not supported
- .idea/misc.xml: Language not supported
- .idea/modules.xml: Language not supported
- .idea/runConfigurations/Sample_1.xml: Language not supported
- .idea/runConfigurations/Sample_2.xml: Language not supported
- .idea/vcs.xml: Language not supported
Comments suppressed due to low confidence (1)
drishti/handlers/handle_darshan.py:572
- The call to sys.exit(2) before invoking the shared small operation check may lead to premature termination of the handler. Verify if this exit is intentional; if not, consider removing it or using proper error handling.
module.check_shared_small_operation(total_shared_reads, total_shared_reads_small, total_shared_writes, total_shared_writes_small, shared_files, file_map)
is_darshan = True and is_darshan | ||
is_recorder = False and is_recorder | ||
else: # check whether is a valid recorder log | ||
if not os.path.isdir(path): | ||
print("Unable to open recorder folder.") | ||
sys.exit(os.EX_NOINPUT) | ||
else: | ||
is_recorder = True and is_recorder | ||
is_darshan = False and is_darshan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The use of expressions like 'True and is_darshan' and 'False and is_recorder' is redundant. Consider assigning the intended boolean value directly (e.g. 'is_darshan = True', 'is_recorder = False') to improve code clarity.
Copilot uses AI. Check for mistakes.
No description provided.